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INTRODUCTION 


This document describes the evaluation of epidemiologic surveillance systems. Its 
purpose is to promote the best use of public health resources through the develop- 
ment of effective and efficient surveillance systems. It can serve as a guide for persons 
conducting their first evaluation and as a reference for those who are already familiar 
with the evaluation process. 

Epidemiologic surveillance is the ongoing and systematic collection, analysis, and 
interpretation of health data in the process of describing and monitoring a health 
event. This information is used for planning, implementing, and evaluating public 
health interventions and programs. Surveillance data are used both to determine the 
need for public health action and to assess the effectiveness of programs. 

The evaluation of surveillance systems should promote the best use of public 
health resources by ensuring that only important problems are under surveillance 
and that surveillance systems operate efficiently. Insofar as possible, the evaluation 
of surveillance systems should include recommendations for improving quality and 
efficiency, e.g., eliminating unnecessary duplication. Most importantly, an evaluation 
should assess whether a system is serving a useful public health function and is 
meeting the system’s objectives. 

Because surveillance systems vary widely in methodology, scope, and objectives, 
characteristics that are important to one system may be less important to another. 
Efforts to improve certain attributes — such as the ability of a system to detect a health 
event (sensitivity)— may detract from other attributes, such as simplicity or timeli- 
ness. Thus, the success of an individual surveillance system depends on the proper 
balance of characteristics, and the strength of an evaluation depends on the ability of 
the evaluator to assess these characteristics with respect to the system’s require- 
ments. In an effort to accommodate to these objectives, any approach to evaluation 
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must be flexible. With this in mind, the guidelines that follow describe many 
measures that can be applied to surveillance systems, with the clear understanding 
that all measures will not be appropriate for all systems. 


ORGANIZATION OF THIS DOCUMENT 


This document begins with an outline of the tasks involved in doing an evaluation, 
which is followed by sections describing each element of an evaluation. The first such 
section addresses the public health importance of the disease or health condition 
under surveillance. The second provides a framework for describing the components 
of a surveillance system. Subsequent sections focus on the attributes of a surveillance 
system (simplicity, flexibility, acceptability, sensitivity, predictive value positive, 
representativeness, and timeliness) and demonstrate how these combine to affect the 
usefulness and cost of a system. The document concludes with a discussion of 
resources needed to operate the surveillance system and sections detailing conclu- 
sions and recommendations. 

As feedback is received on evaluations based on the guidelines given in this 
document, relevant examples will be incorporated into future updates. 


OUTLINE OF TASKS FOR EVALUATING A SURVEILLANCE 
SYSTEM 


A. Describe the public health importance of the health event: The following are the 
three most important categories to consider: 
1. Total number of cases, incidence, and prevalence 
2. Indices of severity such as the mortality rate and the case-fatality ratio 
3. Preventability 


. Describe the system to be evaluated. 
1. List the objectives of the system. 
2. Describe the health event(s) under surveillance. State the case definition for 
each health event. 
3. Draw a flow chart of the system. 
4. Describe the components and operation of the system. 
. What is the population under surveillance? 
. What is the period of time of the data collection? 
. What information is collected? 
Who provides the surveillance information? 
How is the information transferred? 
How is the information stored? 
. Who analyzes the data? 
. How are the data analyzed and how often? 
. How often are reports disseminated? 
j. To whom are reports distributed? 
k. How are the reports distributed? 


‘sro *earce 
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. Indicate the level of usefulness by describing actions taken as a result of the data 
from the surveillance system. Characterize the entities that have used the data to 
make decisions and take actions. List other anticipated uses of the data. 


. Evaluate the system for each of the following attributes: 
. Simplicity 
. Flexibility 
. Acceptability 
. Sensitivity 
. Predictive value positive 
. Representativeness 
. Timeliness 


E. Describe the resources used to operate the system (direct costs). 


. List your conclusions and recommendations. State whether the system is meeting 
its objectives, and address the need to continue and/or modify the surveillance 
system. 


PUBLIC HEALTH IMPORTANCE 
Task 


Describe the public health importance of the health event. 
Definition 


The public health importance of a health event and the need to have that health 
event under surveillance can be described in several ways. Health events that affect 
many people or require large expenditures of resources clearly have public health 
importance. However, health events that affect relatively few persons may also be 
important, especially if the events cluster in time and place—e.g., a limited outbreak 
of a severe disease. At other times, public concerns may focus attention on a 
particular health event, creating or heightening the sense of importance. Diseases 
that are now rare because of successful control measures may be perceived as 
“unimportant,” but their level of importance should be assessed in light of their 
potential to re-emerge. Finally, the public health importance of a health event is 
influenced by its preventability. 


Measures 


Parameters for measuring the importance of a health event—and, therefore, the 
surveillance system with which it is monitored —include: 


1. Total number of cases, incidence,‘and prevalence 
2. Indices of severity, e.g., the case-fatality ratio 
. Mortality rate 
. An index of lost productivity, e.g., bed-disability days 
. An index of premature mortality, e.g., years of potential life lost (YPLL) 
. Medical costs 
. Preventability 


These measures of importance do not take into account the effect of existing 
control measures. For example, the number of cases of vaccine-preventable illness 
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has declined following the implementation of school immunization laws, and the 
public health importance of these diseases would be underestimated by case counts 
alone. In such instances, it may be possible to estimate the number of cases that 
would be expected in the absence of control programs (1 ). 

Preventability can be defined at several levels—from preventing the occurrence of 
disease (primary prevention); through early detection and intervention with the aim 
of reversing, halting, or at least retarding the progress of a condition (secondary 
prevention); to minimizing the effects of disease and disability among those already 
ill (tertiary prevention). From the perspective of surveillance, preventability reflects 
the potential for effective public health intervention at any of these levels. 

Attempts have been made to quantify the public health importance of various 
diseases and health conditions. Dean et a/. describe such an approach using a score 
that takes into account age-specific mortality and morbidity rates as well as health- 
care costs (2). 


SYSTEM DESCRIPTION 


Tasks 


1. List the objectives of the system. 

2. Describe the health event(s) under surveillance. State the case definition for 
each health event. 

3. Describe the components and operation of the system. 

4. Draw a flow chart of the system. 


Methods 


Objectives may include detecting or monitoring outbreaks, monitoring trends, 
identifying contacts and administering prophylaxis, enrolling case-patients in a study, 
and generating hypotheses about etiology. The objectives of the system define a 
framework for evaluating the specific components. 

The next task is to describe the components of a surveillance system. This can be 
done by answering the following questions: 


. What is the population under surveillance? 

What is the period of time of the data collection? 

What information is collected? 

Who provides the surveillance information? What is the data source? 
How is the information transferred? 

How is the information stored? 

. Who analyzes the data? 

. How are the data analyzed, and how often? 

Are there preliminary and final tabulations, analyses, and reports? 
How often are reports disseminated? 

. To whom are reports distributed? 

How are the reports distributed? 


It is often useful to list the discrete steps in the processing of the health-event 
reports by the system, and then to depict these steps in a flow chart (Figure 1). 


rere roam ean fm 
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USEFULNESS 
Tasks 


1. Describe the actions that have been taken as a result of the data from the 
surveillance system. 

2. Describe who has used the data to make decisions and take actions. 

3. List other anticipated uses of the data. 


Definition 


A surveillance system is useful if it contributes to the prevention and control of 
adverse health events, including an improved understanding of the public health 
implications of such events. A surveillance system can also be useful if it helps to 
determine that an adverse health event previously thought to be unimportant is 
actually important. 


Methods 


An assessment of the usefulness of a surveillance system should begin with a 
review of the objectives of the system and should consider the dependence of policy 
decisions and control measures on surveillance. Depending on the objectives of a 
particular surveillance system, the system may be considered useful if it satisfactorily 
addresses at least one of the following questions. Does the system: 


a. Detect trends signaling changes in the occurrence of disease? 
b. Detect epidemics? 


c. Provide estimates of the magnitude of morbidity and mortality related to the 
health problem under surveillance? 
. Stimulate epidemiologic research likely to lead to control or prevention? 
. Identify risk factors associated with disease occurrence? 
. Permit assessment of the effects of control measures? 
. Lead to improved clinical practice by the health-care providers who are the 
constituents of the surveillance system? 


Discussion 


Usefulness may be affected by all the attributes of surveillance. Increased sensi- 
tivity may afford a greater opportunity for identifying epidemics and understanding 
the natural course of an adverse health event in a community. Improved timeliness 
allows control and prevention activities to be initiated earlier. Increased predictive 
value positive enables public health officials to focus on productive activities. A 
representative surveillance system will better characterize the epidemiologic charac- 
teristics of a health event in a defined population. Systems that are simple, flexible, 
and acceptable also tend to be more useful. 


SYSTEM ATTRIBUTES 
Task 
Evaluate the system for each of the following attributes: 


a. Simplicity 
b. Flexibility 





. Acceptability 

. Sensitivity 

. Predictive value positive 
. Representativeness 
g. Timeliness 


A. Simplicity 


Definition 
The simplicity of a surveillance system refers to both its structure and ease of 


operation. Surveillance systems should be as simple as possible while still meeting 
their objectives. 


Methods 

A chart describing the flow of information and the lines of response in a 
surveillance system can help assess the simplicity or complexity of a surveillance 
system. A flow chart for a generic surveillance system is illustrated in Figure 1. 

The following measures might be considered in evaluating the simplicity of a 
system: 


Amount and type of information necessary to establish the diagnosis 
Number and type of reporting sources 
. Method(s) of transmitting case information/data 

Number of organizations involved in receiving case reports 

Staff training requirements 

Type and extent of data analysis 

Number and type of users of compiled case information 

Method of distributing reports or case information to these users 
Time spent with the following tasks: 

1) Maintaining the system 

2) Collecting case information 

3) Transmitting case information 

4) Analyzing case information 

5) Preparing and disseminating surveillance reports 


Discussion 


It may be useful to think of the simplicity of a surveillance system from two 
perspectives: the design of the system and the size of the system. An example of a 
system that is simple in design is one whose case definition is easy to apply and in 
which the person identifying the case will also be the one analyzing and using the 
information. A more complex system might involve some of the following: 


a. Special laboratory tests to confirm the case 

b. Telephone contact or a home visit by a public health nurse to collect detailed 
information 

c. Multiple levels of reporting (e.g., with the Notifiable Diseases Reporting 
System, case reports may start with the doctor who makes the diagnosis and 
pass through county and state health departments before going to the 
Centers for Disease Control) 


Simplicity is closely related to timeliness and will affect the amount of resources 
that are required to operate the system. 


a. 
b. 
c 
d. 
e. 
f. 
g. 
h. 
i. 
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B. Flexibility 


Definition 


A flexible surveillance system can adapt to changing information needs or 
operating conditions with little additional cost in time, personnel, or allocated funds. 
Flexible systems can accommodate, for example, new diseases and health condi- 
tions, changes in case definitions, and variations in reporting sources. 


Methods 

Flexibility is probably best judged retrospectively, by observing how a system 
responded to a new demand. For example, when acquired immunodeficiency 
syndrome (AIDS) emerged in 1981, the existing notifiable disease reporting system of 
state health departments was used to report cases, and AIDS surveillance has 
adapted to rapidly advancing knowledge about the disease, its diagnosis, and its risk 
factors. Another example is the capacity of the gonorrhea surveillance system to 


accommodate special surveillance for penicillinase-producing Neisseria gonor- 
rhoeae. 


Discussion 


Unless efforts have been made to adapt a system to another disease, it may be 
difficult to assess the flexibility of that system. In the absence of practical experience, 
one can look at the design and workings of a system. Generally, simpler systems will 
be more flexible—fewer components will need to be modified when adapting the 
system for use with another disease. 


C. Acceptability 


Definition 


Acceptability reflects the willingness of individuals and organizations to participate 
in the surveillance system. 


Methods 


In terms of evaluating a surveillance system, acceptability refers to the willingness 
to use the system by: a) persons outside the sponsoring agency, e.g., those who are 
asked to do something for the system and b) persons in the sponsoring agency that 
operates the system. To assess acceptability, one must consider the points of 
interaction between the system and its participants (Figure 1), including persons with 
the condition and those reporting cases. 

Quantitative indicators of acceptability include: 


a. Subject or agency participation rates 

b. If participation is high, how quickly it was achieved 

c. Interview completion rates and question refusal rates (if the system involves 
interviews with subjects) 

d. Completeness of report forms 

e. Physician, laboratory, or hospital/facility reporting rates 

f. Timeliness of reporting 


Some of these measures may be obtained from a review of surveillance report 
forms, while others would require special studies or surveys. 





Discussion 

Acceptability is a largely subjective attribute that encompasses the willingness of 
persons on whom the system depends to provide accurate, consistent, complete, and 
timely data. Some factors influencing the acceptability of a particular system are: 


. The public health importance of the health event 
. Recognition by the system of the individual’s contribution 
. Responsiveness of the system to suggestions or comments 
. Time burden relative to available time 
. Federal and state legislative restrictions on data collection and assurance of 
confidentiality 
. Federal and state legislative requirements for reporting 


D. Sensitivity 


Definition 


The sensitivity of a surveillance system can be considered on two levels. First, at 
the level of case reporting, the proportion of cases of a disease or health condition 
detected by the surveillance system can be evaluated. In Table 1 this is represented 
by A/(A+C). Second, the system can be evaluated for its ability to detect 
epidemics (3). 


Methods 
The sensitivity of a surveillance system is affected by the likelihood that: 


a. Persons with certain diseases or health conditions seek medical care; 

b. The diseases or conditions will be diagnosed, reflecting the skill of care 
providers and the sensitivity of diagnostic tests; and 

c. The case will be reported to the system, given the diagnosis. 


These three conditions can be extended by analogy to surveillance systems that do 
not fit the traditional disease care-provider model. For example, the sensitivity of a 
telephone-based surveillance system of morbidity or risk factors is affected by: 


a. The number of people who have telephones, who are at home when the call 
is placed, and who agree to participate; 

b. The ability of persons to understand the questions and correctly identify their 
status; and 

c. The willingness of respondents to report their status. 


The extent to which these questions are explored depends on the system and on 
the resources available for the evaluation. The measurement of sensitivity in a 
surveillance system requires a) the validation of information collected by the system 
and b) the collection of information external to the system to determine the frequency 
of the condition in a community (4). From a practical standpoint, the primary 
emphasis in assessing sensitivity — assuming that most reported cases are correctly 
classified —is to estimate the proportion of the total number of cases in the commu- 
nity being detected by the system. 
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Discussion 


A surveillance system that does not have high sensitivity can still be useful in 
monitoring trends, as long as the sensitivity remains reasonably constant. Questions 
concerning sensitivity in surveil'ance systems most commonly arise when changes in 
disease occurrence are noted. Changes in sensitivity can be precipitated by such 
events as heightened awareness of a disease, introduction of new diagnostic tests, 
and changes in the method of conducting surveillance. A search for such surveillance 
“artifacts” is often an initial step in outbreak investigations. 


E. Predictive Value Positive 


Definition 


Predictive value positive (PVP) is the proportion of persons identified as having 
cases who actually do have the condition under surveillance (5). In Table 1 this is 
represented by A/(A+ 8B). 


Methods 


In assessing PVP, primary emphasis is placed on the confirmation of cases 
reported through the surveillance system. Its effect on the use of public health 
resources can be considered on two levels. At the level of an individual case, PVP 
affects the amount of resources used for case investigations. For example, in some 
states every reported case of type A hepatitis is promptly investigated by a public 
health nurse, and family members at risk are referred for prophylactic treatment with 
immune globulin. A surveillance system with low PVP—and therefore frequent 
“false-positive” case reports— would lead to wasted resources. 

The other level is that of detection of epidemics. A high rate of erroneous case 
reports may trigger an inappropriate outbreak investigation. Therefore, the propor- 
tion of epidemics identified by the surveillance system that are “true epidemics” is 
needed to assess this attribute. 

Calculating the PVP may require that records be kept of all interventions initiated 
because of information obtained from the surveillance system. A record of the 
number of case investigations done and the proportion of persons who actually had 
the condition under surveillance would allow the calculation of the PVP at the level of 
case detection. Personnel activity reports, travel records, and telephone logbooks 
may all be useful in estimating the PVP at the epidemic detection level. 


Discussion 


PVP is important because a low value means that a) non-cases are being 
investigated and b) epidemics may be mistakenly identified. “False-positive” reports 
may lead to unnecessary interventi~.1, and falsely detected “epidemics” may lead to 
costly investigations and undue concern in the community. A surveillance system 
with high PVP will lead to fewer wild-goose chases and wasted resources. 

An example of a surveillance evaluation that examined PVP was reported by 
Barker et al. They reviewed hospital charts to determine the proportion of persons 
admitted with a diagnosis of stroke who had the diagnosis confirmed (6). Of 1,604 
patients admitted to seven acute-care hospitals with a stroke-related diagnosis, 903 
(PVP = 56%) were subsequently confirmed to have had strokes. 

The PVP for a health event is closely related to the clarity and specificity of the case 
definition. Good communication between the persons who report cases and the 
receiving agency also can improve PVP. 
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The PVP reflects the sensitivity and specificity of the case definition and the 
prevalence of the condition in the population (Table 1). The PVP increases with 
increasing specificity and prevalence. 


F. Representativeness 


Definition 
A surveillance system that is representative accurately describes a) the occurrence 


of a health event over time and b) its distribution in the population by place and 
person. 


Methods 


Representativeness is assessed by comparing the characteristics of reported 
events to all such actual events. Although the latter information is generally not 
known, some judgment of the representativeness of surveillance data is possible, 
based on knowledge of: 


. Characteristics of the population—e.g., age, socioeconomic status, geo- 
graphic location (5); 

. Natural history of the condition —e.g., latency period, mode of transmission, 
fatal outcome; 

. Prevailing medical practices—e.g., sites performing diagnostic tests, 
physician-referral patterns (7,8 ); 

. Multiple sources of data—e.g., mortality rates for comparison with incidence 
data, laboratory reports for comparison with physician reports. 


Representativeness can be examined through special studies that seek to identify 
a probability sample of all cases. 

Quality of data is an important part of representativeness. Much of the discussion 
in this document focuses on the identification and classification of cases. However, 
most surveillance systems rely on more than simple case counts. Information 
commonly collected includes the demographic characteristics of affected persons, 
details about the health event, and notification of the presence or absence of potential 
risk factors. The quality and usefulness and representativeness of this information 
depends on its completeness and validity. 

Quality of data is influenced by the clarity of surveillance forms, the quality of 
training and supervision of persons who complete surveillance forms, and the care 
exercised in data management. A review of these facets of a surveillance system 
provides an indirect measure of quality of data. Examining the percentage of 
“unknown” or “blank” responses to items on surveillance forms or questionnaires is 
straightforward. Assessing the reliability and validity of responses would require 
such special studies as chart reviews or re-interviews of respondents. 


Discussion 


In order to generalize findings from surveillance data to the population at large, the 
data from a surveillance system should reflect the population characteristics that are 
important to the goals and objectives of that system. These characteristics generally 
relate to time, place, and person. An important result of evaluating the representa- 
tiveness of a surveillance system is the identification of population subgroups that 
may be systematically excluded from the reporting system. This process allows 
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appropriate modification of data collection and more accurate projection of incidence 
of the health event in the target population. 

For example, an evaluation of reporting of hepatitis in a county in Washington 
State suggested that cases of type B hepatitis were under-reported among homosex- 
ual males and that cases of type non A-non B hepatitis were under-reported among 
persons given blood transfusions. The importance of these risk factors as contribu- 
tors to the occurrence of these diseases was apparently underestimated by the 
selective under-reporting of certain types of hepatitis cases (9 ). 

Errors and bias can make their way into a surveillance system at any stage. 
Because surveillance data are used to identify high-risk groups, to target interven- 
tions, and to evaluate interventions, it is important to be aware of the strengths and 
limitations of the information in the system. 

So far the discussion of attributes has been aimed at the information collected for 
cases, but in many surveillance systems morbidity and mortality rates are calculated. 
The denominators for these rate calculations are often obtained from a completely 
separate data system maintained by another agency, e.g., the Bureau of the Census. 
Thought should be given to the comparability of categories (e.g., race, age, residence) 
on which the numerators and denominators of rate calculations are based. 


G. Timeliness 


Definition 
Timeliness reflects the speed or delay between steps in a surveillance system. 


Methods 


The major steps in a surveillance system are shown in Figure 2. The time interval 
linking any two of the steps in this figure can be examined. The interval usually 
considered first is the amount of time between the onset of an adverse health event 
and the report of the event to the public health agency responsible for instituting 
control and prevention measures. Another aspect of timeliness is the time required 
for the identification of trends, outbreaks, or the effect of control measures. With 
acute diseases, the onset of symptoms is usually used. Sometimes the date of 
exposure is used. With chronic diseases, it may be more useful to look at elapsed time 
from diagnosis rather than to estimate an onset date. 


Discussion 


The timeliness of a surveillance system should be evaluated in terms of availability 
of information for disease control—either for immediate control efforts or for 
long-term program planning. 

For example, a study of a surveillance system for Shigella infections indicated that 
the typical case of shigellosis was brought to the attention of health officials 11 days 
after onset of symptoms—a period sufficient for the occurrence of secondary and 
tertiary transmission. This suggests that the level of timeliness was not satisfactory 
for effective disease control (10). In contrast, when there is a long period of latency 
between exposure and appearance of disease, the rapid identification of cases of 
illness may not be as important as the rapid availability of exposure data to provide 
a basis for interrupting and preventing exposures that lead to disease. In another time 
frame, surveillance data are being used by public health agencies to track progress 
toward the 1990 Objectives for the Nation and to plan for the Year 2000 Objectives. 
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The need for rapidity of response in a surveillance system depends on the nature 
of the public health problem under surveillance and the objectives of that system. 
Recently, computer technology has been integrated into surveillance systems and 
may promote timeliness (177,12). 


RESOURCES FOR SYSTEM OPERATION 
Task 


Describe the resources that are used to operate the system (direct costs). 


Definitions 


This document covers only the resources directly required to operate a surveil- 
lance system. These are sometimes referred to as “direct costs” and include the 
personnel and financial resources expended in collecting, processing, analyzing, and 
disseminating the surveillance data. 


Methods 
In estimating these resources consider the following: 


a. Personnel requirements 
A first step is to estimate the time it takes to operate the system (e.g., 
person-time expended per year of operation). If desired, these measures can 
be converted to dollar estimates by multiplying the person-time by appro- 
priate salary and benefit figures. 
. Other resources 

These may include the cost of travel, training, supplies, equipment, and 
services (e.g., mail, telephone, and computer time). 


The application of these resources at all levels of the public health system —from 
the local health-care provider to municipal, county, state, and Federal health agen- 
cies—should be considered. 


The costs of surveillance systems from two studies are illustrated in Tables 2 and 
3 below (7,13). 


Discussion 


This approach to assessment of resources includes only those personnel and 
material resources required for the operation of surveillance and excludes a broader 
definition of costs that might be considered in a more comprehensive evaluation. 
Estimating the overall costs of a surveillance system can be a complex process. The 
estimates may include the estimation of a) indirect costs, such as follow-up laboratory 
tests or treatment incurred as a result of surveillance; b) costs of secondary data 


sources (e.g., vital statistics or survey data); and c) costs averted (benefits) by 
surveillance. 
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Costs are often judged relative to benefits, but few evaluations of surveillance 
systems are likely to include a formal cost-benefit analysis, and such analyses are 
beyond the scope of this document. Estimating benefits (e.g., savings resulting from 
preventing morbidity through surveillance data) may be possible in some instances, 
although this approach does not take into account the full spectrum of benefits that 
may result from surveillance systems. More realistically, costs should be judged with 
respect to the objectives and usefulness of a surveillance system. 


Examples 


Examples of resource estimation for surveillance systems operated in Vermont 
and Kentucky follow. 


Vermont example (7 ) 

Two methods of collecting surveillance data in Vermont have been compared. The 
“passive” system was already in place and consisted of unsolicited reports of 
notifiable diseases to the district offices or state health department. The “active” 
system was implemented in a probability sample of physician practices. Each week, 
a health department employee called these practitioners to solicit reports of selected 
notifiable diseases. 

In comparing the two systems an attempt was made to estimate their costs. 
The estimates of resources directly applied to the surveillance systems are shown in 
Table 2. 


Kentucky example (73 ) 

Another example is provided by an assessment of the costs of a surveillance 
system involving the active solicitation of case reports of type A hepatitis in Kentucky. 
Table 3 summarizes the costs of this system. The resources that went into the direct 
operation of the system were for personnel and telephone and were estimated as 
$3,764 and $535, respectively. This system found nine more cases than would have 
been found through the passive surveillance system, and an estimated seven 
hepatitis cases were prevented through prophylaxis of the contacts of the nine 
case-patients. 


CONCLUSIONS AND RECOMMENDATIONS 
Tasks 


List your conclusions and recommendations. These should state whether the 
system is addressing an important public health problem and is meeting its objec- 
tives. Recommendations should address the continuation and/or modification of the 
surveillance system. 


Discussion 

The attributes and costs of a surveillance system are interdependent. Before 
recommending changes in a system, interactions among the attributes and costs 
should be considered to ensure that benefits resulting from strengthening one 
attribute do not adversely affect another attribute. 

Efforts to increase sensitivity, PVP, timeliness, and representativeness tend to 


increase the cost of a surveillance system, although savings in efficiency with 
automation may offset some of these costs (12). 
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As sensitivity and PVP approach 100%, a surveillance system is more likely to be 
representative of the population being monitored. However, as sensitivity increases, 
PVP may decrease. Efforts to increase sensitivity and PVP tend to make a surveillance 
system more complex— potentially decreasing its acceptability, timeliness, and flex- 
ibility. For example, a study comparing health-department-initiated (active) surveil- 
lance and provider-initiated (passive) surveillance did not improve timeliness, despite 
increased sensitivity (8 ). 


SUMMARY STATEMENT 


Evaluating surveillance systems is not easy. There is no perfect system; trade-offs 
must always be made. Each system is unique and therefore requires a balancing of 
the effort and resources put into each of its components if the system is to achieve its 
intended goal. 

This document has presented guidelines—not absolutes—for the evaluation of 
surveillance systems. Attributes have been described that can be examined and 
evaluated to assess a system’s ability to achieve the objectives for which it was 
designed. 

Our goal has been to make the evaluation process more explicit and objective. 
Suggestions on how we may improve these guidelines would be welcomed. 
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TABLE 1. Sensitivity and predictive value positive (PVP)*** 
Condition present 
YES NO 


True False 
positive positive 
A B 











Detected by 
surveillance 


False True 
negative negative 
D 





A+C B+D 





*Sensitivity = A/(A+C) 
"PVP =A/(A+B) 
‘Specificity = D/(B+D) 


TABLE 2. Comparison of health-department-estimated costs for active and passive 
surveillance systems, Verment, June 1, 1980-May 31, 1981 (7) 





Type of surveillance system 
Active* Passive’ 
Estimated costs 











Paper $ 114 
Mailing 185 
Telephone 1,947 
Personnel 
Secretary 3,000 
Public health nurses 14,025 


TOTAL $19,271 


*Active = weekly calls from health department to request reports 
*Passive = provider-initiated reporting 








MMWR May 6, 1988 


16 


TABLE 3. Costs of a 22-week active surveillance program for hepatitis A, Kentucky, 
1983 (73) 





Costs 





Activity 
Central office 
Surveillance 
Personnel 
Telephone 


Local health offices* 

Contact tracing 
Personnel 
Telephone 
Travel 

Contact prophylaxis 
Personnel 
ISG’ 


TOTAL 


*Costs of tracing and providing prophylaxis to 38 additional active 
contacts of persons with hepatitis A 
"immune serum globulin 





647 
149 
31 


469 
21 


$5,616 
surveillance-associated 
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FIGURE 1. Surveillance system flow chart 
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FIGURE 2. Simplified flow chart for considering timeliness 
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